Robust Data Partitioning for Ad-hoc Query Processing

Data partitioning can significantly improve query performance in distributed database systems. Most proposed data partitioning techniques choose the partitioning based on a particular expected query workload or use a simple upfront scheme, such as uniform range partitioning or hash partitioning on a key. However, these techniques do not adequately address the case where the query workload is ad-hoc and unpredictable, as in many analytic applications. The Hyper-Partitioning system aims to fill that gap, by using a novel spacepartitioning tree on the space of possible attribute values to define partitions incorporating all attributes of a dataset. The system creates a robust upfront partitioning tree, designed to benefit all possible queries, and then adapts it over time in response to the actual workload. This thesis evaluates the robustness of the upfront hyper-partitioning algorithm, describes the implementation of the overall Hyper-Partitioning system, and shows how hyper-partitioning improves the performance of both selection and join queries. Thesis Supervisor: Samuel Madden Title: Professor